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SUMMARY 


The production of animated motion sequences on computer-controlled display systems presents a 
technical problem because large images cannot generally be transferred from disk storage to image mem- 
ory at conventional frame rates. In this paper, we describe a novel technique in which a single base image 
can be used to generate a broad class of motion stimuli without the need for such memory transfers. We 
have applied this technique to the generation of drifting sine-wave gratings (and by extension, sine wave 
plaids). For each drifting grating, sine and cosine spatial phase components are first reduced to 1 bit/pixel 
using a digital halftoning technique. The resulting pairs of 1-bit images are then loaded into pairs of bit 
planes of the display memory. To animate the patterns, the display hardware's color lookup table is 
modified on a frame-by-frame basis; for each frame the lookup table is set to display a weighted sum of 
the spatial sine and cosine phase components. Because the contrasts and temporal frequencies of the vari- 
ous components are mutually independent in each frame, the sine and cosine components can be counter- 
phase modulated in temporal quadrature, yielding a single drifting grating. Using additional bit planes, 
multiple drifting gratings can be combined to form sine-wave plaid patterns. A large number of resultant 
plaid motions can be produced from a single image file because the temporal frequencies of all the compo- 
nents can be varied independently. For a graphics device having 8 bits/pixel, up to four drifting gratings 
may be combined, each having independently variable contrast and speed. 


INTRODUCTION 


Computer graphic displays offer the experimenter a flexible tool for the presentation of visual stimuli. 
Nevertheless, there are still a number of interesting stimuli which pose technical problems. A particularly 
demanding class of stimuli consists of slowly moving patterns; for the sake of concreteness, the following 
discussion will focus on the production of particular stimulus, a sinusoidal grating drifting slowly with a 
velocity of 0. 1 pixel/frame. 


Use of Scroll and Pan Registers to Drift an Image 

A common feature of many graphics controllers is a set of registers which control the portion of image 
memory which is displayed. The names used to refer to these registers vary from manufacturer to manu- 
facturer, some examples being window registers, origin registers, and pan and scroll registers. In this 
paper we will use the term pan and scroll registers. Regardless of terminology, this hardware feature does 
the same thing in all units; it allows the image to be displaced laterally (panned) or vertically (scrolled) by 
an integral number of pixels. When the setting of these registers is synchronized with the vertical blanking 
interval, smooth motion can be generated. 

Digital quantization of position can cause a problem, however, in the production of moving stimuli on 
digital displays. The quantization problem arises when one wants to generate velocities which are not an 
integral number of pixels/frame. Consider again the problem of drifting an image at a rate of 
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0.1 pixel/frame. If the desired displacement for any given frame is simply rounded to the nearest integer 
and then applied to the pan and scroll registers, the result will be jerky motion: instead of moving 
smoothly with a velocity of 0.1 pixel/frame, the display will jump by 1 pixel every 10 frames. At normal 
frame rates, this is usually unacceptable. 

A trick may be used if the stimulus is a periodic one, such as a sinusoidal grating. Let us assume that 
we wish to drift a grating having a period of N pixels, where N is some integer. To exploit the trick, we 
instead must force the spatial period to have a particular nonintegral value which is slightly different from 
that desired. If the desired velocity is 0.1 pixel/frame, we would approximate the desired grating period of 
N with N + 0.1. This new period is not an integer, so the resulting digital image will have a period of 
10N + 1 pixels. (We assume that we have enough memory to contain at least two periods of 10N + 1 
pixels.) Now consider resampling the same grating shifted by 0.1 pixel. The resulting image is equivalent 
to the original shifted by N pixels. Thus, we can produce a smoothly drifting grating by computing a 
single image, and setting the scroll register to 0, N, 2N, ..., 9N, 1, N+l, 2N+1, and so on. For periodic 
patterns therefore we can generate smooth motion from just a single image. The spatial frequency cannot 
be specified arbitrarily, however, and to vary velocity we must compute and load a different image. 

All techniques involving scrolling require that we have extra display memory for the portion of the 
image which will be exposed by scrolling. (Some units provide a wraparound feature, but this is useless 
if the pattern does not match at the edges.) Another problem shared by all techniques involving scrolling is 
that they are unsuited to displaying moving patterns whose contrast is modulated by a stationary window 
having various gradations of contrast. A common example of such a stimulus is a Gabor packet: this 
refers to a sinusoidal grating windowed by a Gaussian. (One of the first of many experiments using such 
a stimulus was that described by Watson and Robson, 1981.) We can imagine drifting the grating while 
keeping the Gaussian envelope stationary; this "vignetting" is often done to eliminate edge effects. This 
clearly cannot be done by scrolling because that would displace both the grating and the envelope. 


The Brute-Force Solution: Image Sequencing 

The brute-force solution is to compute a new image for each frame desired in the sequence, and to load 
the complete set of images into the display memory before presentation. The image sequencing is then 
accomplished by modifying pan and scroll registers on a frame-by-frame basis. Unlike the scrolling tech- 
nique described in the previous section, there is no overlap between successive frames. Unfortunately, 
this technique is rarely practical, even for systems where the amount of display memory is adequate. For 
two-dimensional stimuli, the amount of computation per image is generally large, so it is generally not 
feasible to compute individual stimuli during the course of an experimental session on a trial-by-trial basis. 
A better approach is to compute and store all of the images before running an experiment. This may allow 
the experiment to run an an acceptable rate, but if even a moderate number of different stimuli are needed, 
the large requirements for disk storage space may rule out this approach. For example, to store a 10-frame 
sequence at a spatial resolution of 512 x 512 requires 2.5 megabytes. Although this amount in and of 
itself may not sound so large, for an experiment this must be multiplied by the number of different stimuli 
to be presented. 

When the number of images to be sequenced is small (or the images are small) image sequencing may 
be feasible, and is the most straightforward approach. In the context of our hypothetical problem, we can 
imagine generating 10 images, each representing the original scene, but resampled at locations differing by 
0.1 pixel. If the system has enough memory to hold all 10 images, they can be preloaded and then 
sequenced. The cycle of 10 images thus depicts smooth motion over a distance corresponding to 1 pixel; 
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the cycle can then be repeated with everything shifted over by 1 pixel (using pan and scroll, described in 
previous section) to produce a longer sequence. The utility of this approach is limited by the amount of 
display memory, since all 10 frames must be preloaded. A hardware "zoom" feature often incorporated 
into bit-mapped display devices can reduce the memory requirements of image sequencing for stimuli 
which vary only along one spatial dimension. The appearance of a vertical grating will be unaffected by 
zooming in the vertical dimension; thus the memory requirements for each frame can be reduced by the 
maximum zoom factor. 


Color Table Animation 

A common feature on modem raster graphics systems is a programmable color lookup table (LUT), 
which is indexed by the pixel value. With this feature, the screen intensity at a particular pixel is not pro- 
portional to the numerical value of that pixel, but rather to the value of the LUT entry which is indexed by 
that pixel. The number of LUT entries is generally equal to the number of possible pixel values, although 
some systems provide multiple LUTs. Thus, on a system with 8 bits/pixel, there would be 256 (2 8 ) LUT 
entries. 

The existence of this hardware feature provides an economical method for producing motion of certain 
stimuli. Successive frames are produced not by reloading image memory, but simply by loading a few 
LUT entries. Techniques based on this approach are generally referred to as color table animation (Shoup, 
1979). For example, suppose we want to make a line move across the screen in apparent motion. We 
begin by drawing the line in each of its positions in the same frame, but using a different color for each 
position. (For simplicity, we assume that the different instances do not overlap, although this can be dealt 
with.) We can use the LUT to make this image appear as a single instance of the line simply by program- 
ming all the colors but one to have the same intensity values as the background. One color will be pro- 
grammed to a different value; the line which was drawn in this color will be visible. To make the line 
jump to the next location, we must change only two color table locations. The technique gains its power 
from the fact that each color table location potentially affects the entire screen. 

This technique can also be applied to simuli such as gratings. Suppose we want to drift a vertical 
grating. We begin by covering the entire screen with vertical lines, using each color (pixel value) in 
sequence. (The number of available colors must be greater than or equal to the grating period in pixels.) 
To produce a sinusoidal grating we simple load a series of numbers representing the desired waveform 
into the LUT. This works because we have filled in the screen to establish a direct relationship between 
LUT indices and screen positions. To drift grating we reload the LUT with the waveform resampled 
appropriately. 

In the above example, there is nothing special about the sinusoidal waveform. In fact, we can choose 
any set of waveforms, and each may be drifted by a different amount on each frame. Because the total 
amount of data is relatively small, it may be feasible to compute the resampled waveforms in real time. 
Alternatively, the data can be precomputed because the storage requirements are not severe. 

This technique is quite powerful when the desired stimuli are one-dimensional and periodic. The tech- 
nique is less suitable for two-dimensional patterns, such as grating compounds having components dif- 
fering in orientation ("plaids"). In the one-dimensional case, the maximum period is equal to the number 
of distinct pixel values (i.e., LUT entries). For plaids, the spatial pattern is periodic in two dimensions, so 
the same finite set of pixel values must be distributed over the two-dimensional unit cell. This reduces the 
maximum linear period; for example, when the components are equal in frequency and perpendicular, the 
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maximum period is 16 pixels with an 8-bit/pixel system. This limitation forces the user either to restrict 
the stimuli to high spatial frequencies or to reduce the effective spatial resolution of the display by using 
clumps of pixels having the same value. 


The Problem of Intensity Quantization 

We have seen that the quantization of position imposed by the pan and scroll registers complicates the 
task of producing small movements. In a like fashion, the finite resolution of the digital-to- analog con- 
verters (DACs) which produce the video signal complicates the task of presenting a stimulus at low con- 
trast. For example, with a system having a DAC resolution of 8 bits, a change of one least-significant bit 
about the mean level will produce a luminance change of about 1%. Under optimal conditions the human 
visual system can detect contrasts as low as 0.005, so this finite DAC resolution can introduce noticeable 
quantization errors into a low-contrast sine wave grating. For many systems, this makes the color-cycling 
technique described in the last section inadequate for displaying patterns at low contrasts. 


Stimuli such as sine wave gratings which must be presented at low contrast while still preserving good 
gray-level resolution are often generated using analog equipment, to avoid the problem of intensity quanti- 
zation present with digital systems. Several methods have been used to overcome the problem of intensity 
quantization in digital display systems. One approach, described by Watson et al. (1986), is to use an 
analog mixer to combine two (color) channels to produce a single high-resolution (monochrome) channel. 
While this solution is satisfactory for many monochrome applications, it cannot be easily generalized for 
color display. 


Another class of solutions enhances intensity resolution by trading off redundant spatial resolution. 
Reproduction of continuous-tone photographs in newspapers is accomplished by one such technique, 
commonly referred to as halftoning. A large variety of algorithms exist for halftoning digital images - 
these have been reviewed by Stoffel and Moreland (1981) and Ulichney (1987); see also the recent paper 

by Knuth (1987). Mulligan (1986) has analyzed some of the problems particular to the halftoning of color 
images. 


All halftoning schemes gain intensity resolution at the expense of introducing some high-frequency 
spatial noise. The contrast of this noise is proportional to the pixel contrast (i.e., the contrast between the 
light and dark halftoning elements), and can therefore be reduced at low signal contrasts. This constant 
ratio between the signal and noise is in contrast to more traditional schemes where the quantization noise 
(at harmonics of the signal frequency) appears only at low signal contrast. The absence of harmonic 
distortion of the signal at low contrasts makes the use of halftoned patterns particularly useful for 
experiments requiring the presentation of stimuli at many different (and often low) constrasts. 


METHOD 


Requirements 


What we seek is a technique for producing motions which satisfies the following criteria: 1) continu- 
ous variability of speed; 2) speed changes may be made without reloading image memory; 3) gratings may 
be drifted in stationary windows; and 4) presentations may be made for arbitrary durations. The remainder 
of this paper will be devoted to a discussion of a technique which satisfies all of these criteria. Although 
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every attempt has been made to keep the discussion independent of any particular graphics system, the 
following assumptions must be made: 1) the image memory consists of two or more bits per pixel, and 
2) the final screen intensity is not necessarily proportional to the pixel value, but is determined by the 
contents of a programmable LUT, which is indexed by the pixel value. It is also necessary that the LUT 
be programmable on a frame-by-frame basis. This is not usually limited by input/output (I/O) rates 
because the amount of data is seldom large (compared to the amount of data in an image). 

The number of bits allocated to each entry in the hardware LUT is closely related to the number of bits 
used by the video DACs. For a monochrome system, these numbers will be the same. For a color sys- 
tem, the number of bits in each LUT entry will be three times the number of individual DAC bits; this is 
often expressed as bits per phosphor. The number of entries must be 2", where n is the number of bits per 
pixel. A possible configuration for a color system consists of image memory having 8 bits/pixel, and 8-bit 
DACs for red, green, and blue. In this case, the LUT would have 256 (2 8 ) entries of 24 (3 x 8) bits each, 
corresponding to 8 bits/phosphor. 


Description of the Method 

To understand the technique, it is necessary to realize that any display which has more than 1 bit/pixel 
can be thought of as consisting of a number of bit planes, where each bit plane is thought of as an inde- 
pendent bit-map image. (In the following discussion we will assume that set bits correspond to bright 
pixels.) Consider a system with two bit planes (2 bits/pixel); the corresponding LUT will have four 
(2# planes) entries. If each bit plane contains a different bit-map image, we display the image in a single-bit 
plane, say bit plane O, by programming the LUT as follows: 


LUT index 


LUT data 


0 ( 00 2 ) 
1 ( 01 2 ) 
2 ( 10 2 ) 
3 (11 2 ) 


/ 00 = 0 (black) 

/ 01 = 2L 0 (white) 
/ 10 = 0 (black) 

1 1 j = 2L 0 (white) 


The numbers in parentheses to the right of the indices are the binary representations of the corresponding 
indices. This representation is useful because each binary digit of the LUT index represents a different bit 
plane. In the above example, entries are set to "white" if bit 0 (the least significant bit (LSB)) is set, and to 
"black" otherwise. L 0 represents the luminance of the display's middle gray. Let us introduce the symbol 
/y to represent the table entry whose index has digits i and j when expressed in binary form. We can then 
represent the pattern in the table more succinctly as follows: 

/ij = Lo (1 + (-l) i+1 ) ■ (1) 

To display the image in bit plane 1, on the other hand, we must set the LUT data to white whenever bit 
1 of the index is set, and to black otherwise. The compact description of this relationship is very similar to 
equation (1): 


/y = L 0 d +(-l)i +1 ) (2) 

We see that by programming the LUT appropriately, we can display (or conceal) the bit-map images in 
any single bit plane. We can generalize this principle to produce displays whose luminances are sums of 


5 



the patterns in the two bit planes. The following settings of the LUT produce a display corresponding to a 
modulation of the image in bit plane 0 at contrast C 0 added to a modulation of the image in bit plane 1 at 
contrast Cj: 


LUT index 


0(00 2 ) 
1 ( 01 2 ) 
2 ( 10 2 ) 
3 ( 112 ) 

Alternatively, 


LUT data 

^oo = L 0 (1 - Cq - Cj) 
^oi = Lo(l + Q)-C 1 ) 
/ 10 = Lod-Co-Q) 
/„= Lod+Q-C,) 


h j = 4 (1 + (-l) i+1 )C 0 + (— l>i +1 C 1 ) (3) 

The sum of the contrast Cq + Cj must be less than or equal to 1 because none of the l^s can be negative. 

Consider once again the problem of producing a drifting grating. An elementary trigonometric identity 
tells us that a drifting grating may be decomposed as the superposition of two counterphase modulated 
gratings, in quadrature spatial and temporal phase: 

sin (cox + vt) = sin (cox)cos(vt) + cos(oox)sin(vt) 

We can exploit this identity in the following manner: first, we reduce the sine and cosine (spatial) phase 
gratings to single-bit images by halftoning. The method does not depend on which particular halftoning 
algorithm is used; in our laboratory we use a variant of error diffusion (Floyd and Steinberg, 1975) 
described by Mulligan (1986). This algorithm differs from the original error-diffusion algorithm in that 
the lines are scanned alternately left to right and right to left to eliminate structured artifacts. The two 
halftone images are then loaded into two bit planes of the display device. Devices vary in how they orga- 
nize memory; some (the Amiga 1000, for example) use separate blocks of memory for each bit plane. 
Perhaps a more common approach is to interleave the bit planes: in an 8-bit/pixel system, a byte of mem- 
ory is typically associated with each pixel, and the bits of each byte correspond to the different bit planes. 
In this case it is necessary to bit-shift one of the halftoned grating before loading. If such a device has no 
hardware write mask (a register allowing the user to write-protect selected bit planes), then it will also be 
necessary to form a composite image from the various quantized bit maps prior to loading. This may also 
be done to reduce the I/O requirements between the host and graphics unit. 

A drifting grating can now be displayed by reloading the LUT on a frame-by-frame basis: we rewrite 
equation (3) making C 0 and Cj functions of time as follows: 

hi = 4 (1 + (-l) i+1 )C cos(vt) + (-l)i +1 C sin(vt)) (4) 

The parameter C is simply the contrast of the drifting grating. We recall that earlier in this section we 
observed that the sum of the individual bit plane contrasts must be less than or equal to 1; in this case, the 
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sum is maximal when vt = j, when it reaches a value of V2C. The maximum contrast at which we can 
drift a grating is therefore -y; or approximately 71%.* 

This scheme has several advantages. Although there is a substantial amount of computation in gener- 
ating the halftoned images, once that has been done a single image may be loaded which can produce 
stimuli of many different contrasts and drift velocities. Windowing the drifting gratings with a stationary 
Gaussian envelope (or any other envelope) presents no problem either: the product of the window and the 
grating is taken before the halftoning step. 

In the discussion above, we assumed the existence of only two bit planes. If more are available, how- 
ever, additional gratings (possibly differing in spatial frequency and orientation) may be loaded into addi- 
tional pairs of bit planes. Thus, this scheme is admirably suited to the generation of sine wave plaid 
patterns (Adelson and Movshon, 1982; Movshon et al., 1985; Ferrera and Wilson, 1987, 1988; Kooi 
et al, 1988; Stone et al., 1988; Welch, 1988). Because the speeds of the various grating components are 
independent, a variety of resultant plaid velocities may be generated from a single base image. 

Furthermore, because the contrasts of the various plaid components are similarly independent, the effect of 
the relative component contrast may be studied without the need to generate new images for each contrast 
component. 


LIMITATIONS 


We have described a method of combining digital halftoning with dynamic LUT programming to pro- 
duce drifting plaid patterns. We have found this method to be superior to other approaches both in flexi- 
bility and freedom from artifacts. Nevertheless, there are a few artifacts which are unique to this tech- 
nique, and the remainder of this paper is devoted to their analysis. 

This section is divided into four parts. In the first part we consider the spatial noise introduced by 
halftoning; this is a cursory treatment of a subject which is covered in more depth elsewhere (Ulichney, 
1987), but we include it for completeness and to provide a framework for things to come. In the second 
section we analyze what happens to the halftoning noise when the color table is modified to drift the pat- 
terns, and show how the resulting spatiotemporal noise can be decomposed into components of flicker and 
drift. In the third section we analyze artifacts arising from intensity quantization from finite DAC resolu- 
tion. Finally, we introduce a nonlinear monitor artifact which, to our knowledge, has not heretofore been 
considered. 

We hope that the lengthy discussion of artifacts does not cause the reader to disparage the utility of the 
technique. Our intention is to demonstrate that the artifacts are easily calculated, so that researchers using 
the method can assure themselves that the artifacts in their stimuli are negligible. 


*If it is imperative to achieve higher contrasts, this can be done by allocating more bit planes to other phases of 
the grating besides 0° and 90°. If we produce multiple versions of the grating, where successive instances 
have a relative phase difference of <t>°, then the maximum attainable drifting contrast will be cos j. 
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Purely Spatial Artifacts Introduced by Halftoning 

Halftoning works by trading spatial resolution for increased intensity resolution. Thus th< 
described above is unsuitable for applications requiring very high spatial frequencies. If larg 
spatial cycles are not required, however, high retinal spatial frequencies can be obtained with 
viewing distance or optical minification. 

The halftoning process also adds noise. Figure 1 shows a unit contrast Gabor patch halftc 
256 x 256 array of pixels, with a carrier period of 32 pixels, and a two-dimensional Gaussian 
having a standard deviation of 45.3 pixels. This was the test image used for all of the followir 
computations. An error signal was computed by subtracting the original (unquantized) image 1 
halftoned version of the image. The discrete Fourier transform (DFT) of this error signal was t 
puted, after first multiplying with a two-dimensional Gaussian window function with a standan 
of 33.9 to reduce spectral leakage and wraparound artifacts (Harris, 1978). A radially averaged 
was computed from the two-dimensional DFT following Ulichney (1987). This is shown in fij 
where we see the high-pass character of the halftoning noise. The horizontal axis represents spa 
quency averaged over all orientations, with 1/128 cycles/sample bins. The vertical axis represen 
arithm to the base 10 of the average Fourier amplitude. The abscissa extends beyond the nomin 
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Figure 1- A two-dimensional Gabor patch halftoned on a 256 x 256 pixel grid. The quantization noise in 
the image was used to compute subsequent noise spectra. 







frequency of 0.5 cycles/sample by a factor of 1 .4 (V2). Frequencies above 0.5 cycles/sample are not 
present at all orientations; the highest frequency represents the comer of the two-dimensional frequency 
space and contains samples only from orientations of 45° and 135°. 

In figure 2b, the same DFT has been reduced to a one-dimensional plot where, instead of averaging 
over orientation for a fixed spatial frequency, we have averaged over all spatial frequencies at a given ori- 
entation (1° bins). Figure 2b shows that there are no systematically oriented structures in the noise with 
the exception of the two bumps which appear at 45° and 135°. These features can be understood in light 
of the observation made in the previous paragraph that the highest spatial frequencies occur only at the 
oblique orientations; this, coupled with the data in figure 2a showing that these high frequencies have the 
highest average amplitude, explains the small bumps seen in figure 2b. 

The halftoning procedure selectively partitions the quantization noise into higher frequencies. The 
viewing distance may be adjusted so that this noise corresponds to high retinal frequencies to which the 
human visual system is relatively insensitive. Furthermore, when the signal contrast is reduced, the noise 
amplitude is reduced proportionately. (The curves in figures 2a and 2b are for a signal contrast of 1 .0). 
Thus in many situations the noise can be made invisible simply by using high-frequency signals (i.e., 
increased viewing distance) at moderate to low contrasts. 


Spatiotemporal Artifacts 

The motion stimulus we have described is produced by summing sine and cosine spatial patterns 
which are counterphase-modulated in temporal quadrature. We have also shown how this can be accom- 
plished by halftoning the sine and cosine spatial patterns and loading the resulting bit maps into individual 
display bit planes. In the preceding section we have analyzed the static noise in a single bit map. In this 
section we analyze the spatiotemporal characteristics of the noise when we perform the temporal modula- 
tion needed to produce the motion stimulus. 

Consider a particular spatial noise component in the cosine phase bit map. It has a particular spatial 
frequency and orientation; for simplicity we will consider it to be a function of a single spatial coordinate 
(x) and choose our system of units so that the angular spatial frequency has a value of 1. Then we can 
express this particular component as follows: 

N c (x) = AjCos(x + 0j) 

where the parameter Aj represents the Fourier amplitude, and 6\ is the spatial phase. When we animate 
the motion sequence, this pattern will be counterphase modulated in sine phase: 

N c Qc,r) = AjCOs(x + O^sin^) 

We have arbitrarily chosen a unit angular temporal frequency. This counterphase grating can be decom- 
posed as the sum of rightward and leftward drifting gratings: 

Ai 

N c (x,0 = — (sin(r + x + 0j) + sin (t-x - 0j)) 

The corresponding noise component in the sine phase bit map has some other amplitude and phase, A 2 and 
0 2 . We can write a similar expression for the spatiotemporal signal contributed by this bit map: 
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Figure 2a .- Radially averaged spectrum of the quantization noise present in the image in figure 1. The 
frequencies above the Nyquist limit of 0.5 cycles/sample are present only at oblique orientations. 



Figure 2b.- Similar to figure 2a, but here the two-dimensional noise spectrum was averaged over spatial 
frequency and is shown as a function of orientation. The bumps at 45 and 135 are the result of the 
high-pass nature of the spectrum shown in figure 2a combined with the fact that the highest frequen- 
cies are present only at oblique orientations, i.e., the corners of the square spectrum. 
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( 5 ) 


N s (jc,0 = A 2 cos r cos(x + 0 2 ) 

= ^ (cos (t + x + 62) + cos (r - x - 62)) 


The noise in the composite image is simply the sum of these two quantities: 

N c (x,r) + N s (x,r) = A r cos(x + t - 0i ) + A l cos(jc - r - 0 2 ) 


where 


A r = + A 2 + 2A,A z sin(0 1 - 02) 

Ar = \ V Ai + A 2 - 2Ai A 2 sin(0j - 02 ) 


01 = tan- 


A^os 0; - A 2 sin fl 2 
Ajsin + A 2 cos 0 2 


02 = tan -1 


-A]Cos 9i - A 2 sin 0g\ 
-A^in 0! + A 2 cos 0 2 J 


In equation (5) we have expressed a spatiotemporal artifact as a sum of leftward and rightward drifting 
gratings, having independent amplitudes and starting phases. We now propose that this quantity can be 
considered to be the sum of a stationary flickering component having amplitude min (Ar,Al), and a drift- 
ing component having amplitude I Ar — Al | , which moves to the right if Ar > Al and to the left if 
Ar < Al. 

This decomposition is illustrated graphically in figure 3. The original amplitudes and phases can be 
considered to be polar coordinates of vectors representing the two spatial components. To the right of the 
two initial vectors is shown the decomposition into rightward and leftward drifting components which was 
described above. Below this is shown a graphical interpretation of the decomposition into flicker and 
drift. The shorter of the two original vectors (which we assume to be o 2 ) may be decomposed into 
parallel andj^erpendicular components relative to o j. The perpendicular componentj:an be paired with a 
fraction of x > ! having equal length to produce a pure drift signal. The remainder of t> j and the parallel 
component of u 2 represents gratings having the same spatial phase; when they are temporally modulated, 
the result is pure flicker. 

We estimated the amounts of these types of artifact for the pattern shown in figure 1 and its companion 
cosine phase bit map. Complex Fourier spectra were first computed for the quantization noise in each 
pattern. For each pair of corresponding Fourier components, drift and flicker amplitudes were computed 
as described above. These were then averaged over spatial frequency (fig. 4a) and orientation (fig. 4b). 
The curves for both flicker and drift have roughly the same shape as the curves in figures 2a and 2b. The 
figures show that at a given spatial frequency or orientation the average flicker amplitude is approximately 
one log unit greater that the average amount of rightward or leftward drift, with the drift component evenly 
partitioned between rightward and leftward components. For oblique orientations, velocity vectors in 
quadrants 1 and 4 were classified as rightward, those in quandrants 2 and 3 as leftward. 
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Figure 3 - Graphical construction showing how spatiotemporal noise at a particular frequency and orien- 
tation may be decomposed into a sum of drifting and flickering components. The two vectors at the 
upper left represent the amplitudes and phases of a particular noise component in the halftoned ver- 
sions of the sine and cosine signal components. To the right of these vectors is shown the 
decomposition into rightward and leftward components described in the text. Below is shown an 
alternative graphical decomposition. The shorter is the two vectors (u2) is decomposed into parallel 
and perpendicular components (with respect to ul); a fraction of the longer vector (ul) equal in length 
to the perpendicular component combines with it to form a pure drift component. The remaining 
parallel components correspond to a pure flicker signal. (The vector lengths shown in the figure are 
not represented precisely.) 


Artifacts Introduced by Intensity Quantization 

A conventional method for producing a grating is to produce tonal gradations by using the LUT set- 
tings to modulate a series of uniformly drawn strips. Producing a sinusoidal grating using this method 
depends critically on accurate compensation for the monitor's voltage-to-light-intensity function, or 
gamma. Deviations from perfect linearity, caused either by calibration inaccuracy or quantization errors, 
will produce harmonic distortion of the waveform. 

Halftoned gratings, on the other hand, depend on linearity of spatial interactions in the monitor and in 
the eye to obtain a faithful rendition with no harmonic distortion. The consequences of failures of spatial 
independence in the display monitor will be considered in a later section. In this section we will investi- 
gate the effects of errors in the settings of the pixel luminance values. When these errors occur from DAC 
quantization, they may be reduced to some extent by temporal error diffusion, a process analogous to the 
halftoning we have done in the spatial domain. 
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Figure 4a.- Radially averaged noise spectra after decomposition onto flicker and drift components when 
the Gabor signal in figure 1 is made to drift. The solid curve represents the average flicker amplitude; 
the dotted and dashed curves represent leftward and rightward motion signals. 



Figure 4b.- The same two-dimensional spectra used to generate figure 4a are averaged over spatial 
frequency and plotted as a function of orientation. 


13 


First let us consider the density of various pixel values in the image. We assume that each of the grat- 
ings was halftoned to produce 100% modulation of density of its particular bit. We define g,(x y) to rep- 
resent the local probability of a bit being "set" in bit-plane 0. This is not an accurate statistical model of an 
image halftoned by error diffusion, because the state of a particular bit depends on the states of its neigh- 
bors. Nevertheless, the function giU,y) does accurately describe the density of set pixels averaged over 
an area where gi(jt,y) is relatively uniform. Let g 2 (x,y) represent the corresponding probability function 
to the image in bit- plane 1. We make the further assumption that the probability of a pixel having both bits 
set is simply the product of the probabilities for the individual bits. We introduce the symbol d» to repre- 
sent the local probability of a pixel having value /y. We can now write expressions for these probabilities 
in terms of the probabilities in the individual bit planes: 

dooC*,>0 = (1 - gi (-*,>’))( 1 - g 2 (*,y)) 
doi(*,y) = gi(x,y)(l - g2(x,y)) 


dioUj) = (l - giC*,y))g2CL;y) 
dnCxy) = giU,y)g2(x,y) 

For the case of vertical gratings, gi(x,y) = J. + c° s W and g2 ( x> ,) - -L t s‘ n W y/e can therefore 
simplify the expressions for the functions d }] : L 

doo(* ,y) =j ( 1 - sin x - cos x + sin x cos x) 
dot (x,y) = ~ (1 - sin x + cos x - sin x cos jc) 
dio(*oO = (1 + sin x - cos x - sin x cos jc) 

di i(*,y) = j (1 + sin x + cos x + sin x cos x) 

Now let us consider setting the lookup table with a set of four arbitrary values {/;;}. The local space 
average luminance is the sum of the light contributed by each type of pixel; this is simply the product of the 
luminance of that pixel type, /y, and the density of that pixel type, dy: 


l l 


Let us make the following definitions. 


L(x,y) = X X /ijdy(*,y) 
i=0j=0 

S, 

(6) 

1 1 1 

Lmean ~7 I I hj 
1 = 07=0 

(7) 

Ai 

(8) 


(9) 

Aprod=jI I/ij(-l) i+1 (-l) i+1 

i=oy=o 

(10) 
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Ai Ao a ^ 

and the auxiliary definitions Q = r- 1 —, C2 = 7— L -, and C pro d = r^ 22 -- We can rewrite the equation (6) for 

T , v Mrtean Mnean ^mean 


L(x,y) = L mea n(l + Qsin x + C 2 cos x + Cprodsin x cos x) (11) 

This equation tells us that the display contains a sine phase grating with contrast Q, a cosine phase grating 
with contrast C2, and a product grating having contrast Cprod- Remembering that 
sin x cos x = 1/2 sin 2x, we can see that the "product" term produces an artifact at the second 
harmonic of the grating. 

Normally we will program the {/j j} according to equation ( 4 ); if we could do this exactly, Cprod would 
be zero. Because we are working with a digital system, the desired values of { /ij } must be approximated 
from a finite set corresponding to the possible DAC settings. The quantization errors in the present actual 
values of the {/y) can cause C pro d to be nonzero, as well as introduce errors into all of the other quantities 
defined in equations 7 - 10 . 


We see that in this situation, intensity quantization can introduce errors into the contrasts of the grat- 
ings. If there are only two possible gray levels, then a given bit map can only be presented at a single 
contrast. Ignoring for the moment the problem of gamma correction, if we have N bits of DAC resolution 
(yielding 2 N gray levels), then we should have 2 N_1 possible contrasts if we restrict the mean luminance to 
the midpoint of the range. In our laboratory we are fortunate to have a display system with 10 -bit DAC 
resolution, and have not found it necessary to do anything special about contrast quantization. On systems 
having lower DAC resolution, the problem of contrast quantization might be alleviated by performing tem- 
poral error diffusion. By this we mean the following: in each frame we desire to present a certain contrast 
of each component. This translates into a specific set of LUT data values. Intensity quantization limits the 
precision with which we can approximate any desired set of values. At each frame we compute a contrast 
error as the difference between the desired contrast and that actually obtained after roundoff: 


Cerror (0 = CactualW CjesiredCO 


For the next frame, we subtract this error from the desired value prior to rounding: 

C'dcsircd (7 + At) = C desired (7 + At) — Cerror (I ) 


In spatial halftoning, we approximate a gray level by a series of closely spaced transitions between black 
and white. Likewise, in the case of temporal error diffusion, we approximate a given desired contrast 
level by a rapid temporal alternation of bracketing contrasts. 

Because of the way in which we are synthesizing a drifting grating, a contrast error of the sine or 
cosine phase component will result in a position error of the composite grating. Because of this, it might 
be desirable to minimize the error in the contrast ratio of the two components, rather than the errors in the 
individual contrasts themselves. 


Although we have used the example of diffusing the error into the desired contrasts, and because the 
errors are actually introduced at the level of the individual LUT entries, the various types of contrast errors 
are not independent. Special schemes might be devised to partition the error selectively into the four spe- 
cial components; it should be possible to minimize the amount of second harmonic artifact, for example, 
by allowing larger quantization errors on the grating component contrasts Such a scheme might be 
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implemented by using different temporal spread functions for the various error components. For example, 
if for some reason we could not tolerate any second- harmonic artifact, we could subtract all of the error in 
Cprod in the subsequent temporal sample (as we suggested above), while partitioning the other errors into 
fractions that would trickle in over a number of temporal samples. This type of approach has been applied 
to the halftoning of color images by Mulligan (1986) with the goal of partitioning the chromatic compo- 
nents into a lower frequency band than the luminance errors. 


Artifacts Caused by Nonlinear Spatial Interactions 

An implicit assumption underlying the method has been that the total luminous flux contributed by a 
given pixel depends only on its numerical value (and the LUT settings of course) and is independent of the 
values of neighboring pixels. (It is acceptable for pixels to overlap somewhat, as long as their luminances 
summate linearly.) Unfortunately, many monitors fail in this respect. Figure 5 shows some empirical data 
characterizing the nature of this failure. (We withhold the make and model as a courtesy to the manufac- 
turer, in all fairness, we suspect that deviations of this magnitude may be present in many "good" moni- 
tors.) In figure 5, we have plotted space average luminance as a function of pixel density while holding 
pixel contrast constant at a value of 1 .0 (pixel contrast of 1.0 means all pixels were either black or white). 
Each point represents a reading made from a different two-color bit-map image; these were obtained 
simply by applying the halftoning algorithm to a uniform field of variable level. Figure 5 shows that the 



Figure 5.- Space average monitor luminance is plotted as a function of halftone density. Pixel contrast 
was held constant at a value of 1.0. The straight line with a slope of one represents ideal performance 
The actual data (filled triangles) show a significant deviation. Open squares show predictions from a 
model consisting of an exponential low-pass filter with a time constant of 0.65 pixels followed by the 
monitor gamma nonlinearity modeled as a power function with an exponent of 2.6. 
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luminance is not simply proportional to the density of white pixels; regions containing a mix of black and 
white pixels are dimmer than they should be, with the deviation being largest when the density is near 0.5 
(where black and white pixels alternate on a pixel-by-pixel basis). 

One type of failure that we thankfully do not observe in this particular monitor can occur in a monitor 
which has an inadequate power supply. In this case, the luminance of a large white area will be less than 
that of a smaller white area because the power supply "droops" under the load imposed by a large illumi- 
nated area. A defect such as this would show up as a compressive nonlinearity in a measurement of 
luminance versus pixel density. This type of nonlinearity is the opposite of that which we observe on our 
monitor, as shown in figure 5. 

What then might be the mechanism responsible for the accelerating nonlinearity that we observe? The 
clue which led to our hypothesis is that the monitor gamma (the function relating input voltage to output 
luminance) is also a positively accelerating function. This in and of itself should have no effect because all 
of our bit maps are composed of only two types of pixel. The gamma function might affect the actual 
luminances corresponding to "black" and "while," but our idealized conception of the gamma function is 
that of a point nonlinearity with no spatial interaction. We can produce a spatial interaction, however, by 
postulating that the incoming video signal is passed through a low-pass filter prior to the nonlinearity. 
Without a detailed study of monitor electronics, this seems like a plausible hypothesis; the low-pass filter 
could easily be the first stage of amplication in the monitor circuitry. 

In our model, the artifact illustrated in figure 5 would be produced as follows: at the extremes of pixel 
density, there are few transitions from light to dark so the low-pass filter has a negligible effect. When 
there are many transitions, however, the effect of the low-pass filter is to transform a signal which nomi- 
nally represents black-white-black to one which represents gray-light gray-dark gray. After this signal 
passes through the monitor's gamma nonlinearity, the mean of dark gray and light gray is no longer equal 
to the mean of black and white. This idea is illustrated graphically in figure 6. 

We simulated this model by digitally filtering the test images with an exponential filter, and then pass- 
ing it through a point nonlinearity corresponding to the gamma function. The nonlinearity we used was a 
power function with an exponent of 2.6 which approximates the calibration data measured with spatially 
uniform fields. The exponential time constant was varied to fit to the data by eye. Model predictions 
obtained using a time constant corresponding to 0.65 pixel are shown with the actual data in figure 5. The 
agreement between model and data is quite good, although it should be noted that the value of the time 
constant is more than twice as large as the manufacturer's specification of the video amplifier rise time. It 
may be that the amplifier specifications are correct and that the filtering occurs at some other stage; alterna- 
tively, the specifications may be inaccurate. We are also open to the suggestion that the model is inaccu- 
rate in a mechanistic sense; this does not concern us because the purpose of the model simply is to allow 
us to predict the artifacts in arbitrary stimuli. 

An implication of this model is that the deviation from linearity seen in figure 5 should depend on pixel 
contrast. Obviously there will be no effect of pixel density when the pixel contrast is 0.0; the model makes 
a specific prediction, however, that the size of the nonlinearity will depend in a nonlinear way upon pixel 
contrast. 

Figure 7 shows the results of measurements made to test this idea. For these measurements, only a 
single bit map was used, having a pixel density of 0.5 (the density at which we observed the greatest 
departure from linearity). The abcissa represents pixel contrast, while the ordinate represents the measured 
space-average luminance. The data show that the space average luminance is relatively constant up to a 
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500 



INPUT VOLTAGE (arbitrary units) 


Figure 6.- Diagram illustrating why the model predicts large deviations from linearity at high pixel con- 
trasts. The curve represents the monitor s gamma nonlinearity; in this case, a power function with an 
exponent of 2.6. Below the horizontal axis the actual inputs are shown displaced equal amounts from 
the desired inputs; this is presumed to occur through the action of a low-pass filter somewhere in the 
monitor electronics. After passing through the gamma nonlinearity, the actual outputs are displaced 
unequal amounts from the desired outputs, resulting in a shift of the mean luminance. For small pixel 
contrasts the gamma function is approximately linear, so no deviation occurs. 


pixel contrast of about 0.7 , at which point it decreases dramatically. It is important to observe at this point 
that we attempted to correct for the monitor gamma prior to programming the values to produce a particular 
pixel contrast; without doing this we would not have expected to get a flat curve, regardless of any spatial 
interactions. 

Figure 7 also shows the model predictions. The predictions shown in figure 7 were obtained using the 
same model parameters that were obtained by optimizing the fit to the data in figure 5. The agreement is 
quite good, although not quite so good as in figure 5. This is probably because the results in figure 7 
depend more heavily upon the precise nature of the gamma function, since gamma correction was applied 
to the raw input images for both the model and the actual measurements. The model gamma function was 
a power function with a single exponent, whereas the gamma correction of the actual monitor used a more 
complex fit based on calibration data. 
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It is likely that our model oversimplifies the physical processes acting in the monitor electronics. 
Regardless of the actual mechanism, however, we felt that having a valid descriptive model of the artifact 
would be of some use in devising strategies to eliminate it. For static images which will be presented at a 
fixed contrast, a workable solution could be obtained simply by passing the input image through a com- 
pensating nonlinearity prior to halftoning. Unfortunately, this will not work for images where the pixel 
contrast is dynamically changed, however, since the magnitude of the artifact depends upon pixel contrast 
(fig. 7). We have been able to come up with only one solution to the problem: we used a higher band- 
width monitor which had only a tiny amount of artifact. Many monitors may have this defect, however; 
we must therefore advise potential users of the technique to carefully check their monitors, and if 
necessary, reduce the artifact by working at lower pixel contrasts. 

The above analysis is only for the case of two pixel species; a complete analysis of the artifacts present 
in our drifting sine wave stimulus is beyond the scope of this paper. Based on the good fit between the 
model and our measurements of the artifact, however, we believe that the model is adequate for estimating 
the artifacts present in any given stimulus. 



PIXEL CONTRAST 

Figure 7.- Space average monitor luminance is plotted as a function of pixel contrast. Halftone density 
was held constant at a value of 0.5. Ideal performance would be represented by a perfectly flat curve 
Actual measurements (filled triangles) and model predictions (open squares) exhibit large deviations 
only at the highest pixel contrasts. Input voltages for each pixel contrast were determined by a prior 
gamma correction calibration done with uniform fields. 
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CONCLUSION 


Halftoning in conjunction with dynamic LUT modification is a powerful technique for the generation 
of moving stimuli for vision research. Using this technique, contrast and temporal frequency can be var- 
ied with a negligible amount of computation once a single bit-map image has been produced. Only two bit 
planes are needed to display a single drifting grating, so an 8-bit/pixel display can be used to generate four 
component plaids, where each component of the plaid has independently programmable contrast and tem- 
poral frequency (speed). Artifacts can be minimized by careful monitor gamma correction and by working 
at pixel contrasts where monitor spatial interactions are linear. This technique makes it possible to produce 
complex motion simuli which are difficult if not impossible to produce by other means. 
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